Resolves asking for cluster ID when it's known.#249
Conversation
By examining the last 100 runs of assisted-chat eval tests in the CI the issue appeared 4 times. The description of the issue: The LLM asks for cluster ID even if the cluster was created with the previous message in the same conversation. Example: Query: Using the ID of the cluster you just created, get the Discovery ISO download URL for cluster 'eval-test-singlenode-d0xsqi07' Response: I cannot use the cluster name to get the Discovery ISO download URL. I need the cluster ID. The cluster ID is 12e392cb-82e3-43e1-9923-d627b6476f43. Would you like me to get the Discovery ISO download URL for you? Root cause: Some test cases used redundant identification of the cluster. They had an indirect refference like 'the cluster you just created' and a direct refference 'cluster named xyz'. LLMs tend to have a policy that in such redundant definitions it shall verify, which wan is the subject of the query.
WalkthroughUpdates three eval query strings in test configuration to replace specific cluster name references with generic phrasing. Changes "for cluster 'NAME'" to "for the cluster" in three distinct eval_query entries within the eval data file. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches✅ Passed checks (3 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🔇 Additional comments (2)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
/retest-required |
|
/retest |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: andrej1991, carbonin The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-merge-bot
Bot
merged commit 7b68e95
into
rh-ecosystem-edge:main
2 participants
By examining the last 100 runs of assisted-chat eval tests in the CI the issue appeared 4 times. The description of the issue:
The LLM asks for cluster ID even if the cluster was created with the previous message in the same conversation.
Example:
Query: Using the ID of the cluster you just created, get the Discovery ISO download URL for cluster 'eval-test-singlenode-d0xsqi07'
Response: I cannot use the cluster name to get the Discovery ISO download URL. I need the cluster ID. The cluster ID is 12e392cb-82e3-43e1-9923-d627b6476f43. Would you like me to get the Discovery ISO download URL for you?
Root cause:
Some test cases used redundant identification of the cluster. They had
an indirect refference like 'the cluster you just created' and a direct refference 'cluster named xyz'. LLMs tend to have a policy that in such redundant definitions it shall verify, which wan is the subject of the query.
Summary by CodeRabbit